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Executive  Summary 


The  Advanced  Distributed  Learning  (ADL)  Initiative  was  established  by  the  Office 
of  the  Undersecretary  of  Defense  for  Personnel  and  Readiness  (OUSD(P&R))  in  1997.  Its 
purpose  was  to  assist  the  military  Services  in  making  “learning”  (education,  training,  and 
performance/decision  aiding)  available  on-demand,  anytime  and  anywhere. 

“Learning”  in  ADL  is  used  as  a  catch-all  designator  for  education,  training,  and 
performance/decision  aiding.  “Distributed”  in  ADL  signifies  learning  that  can  be 
provided  in  classrooms  with  a  teacher  present,  in  the  field  linking  together  widely 
dispersed  instructors  and  students,  or  standing  alone  with  no  instructor  other  than  the 
computer  itself.  “Advanced”  in  ADL  implies  affordable,  interactive,  adaptive,  on-demand 
instruction  delivered  using  computer  technology  so  that  it  is  available  anytime,  anywhere. 

Empirical  research,  available  as  early  as  the  1960s,  suggested  the  feasibility  of  ADL 
and  its  goals.  It  has  shown  that: 

•  Individualized,  tutorial  ‘learning’  (including  individualized  performance/deci¬ 
sion  aiding)  can  be  provided  affordably  by  technology-based  learning. 

•  Technology-based  learning  can  be  more  effective  and  can  produce  greater  return 
on  investment  than  conventional  instructional  approaches  across  many  instruc¬ 
tional  objectives  and  subject  matters. 

•  Technology-base  learning  allows  education,  training,  performance  aiding,  and 
decision  aiding  to  be  delivered  on  platforms  ranging  from  hand-held  devices,  to 
desk-top  computers,  to  capabilities  embedded  in  operational  equipment. 

Statistical  findings  from  this  research  may  be  summarized  by  a  “Rule  of  Thirds.”  It 
states  that  application  of  technology-based  learning  can  reduce  the  cost  of  instruction  by 
about  one-third.  Additionally  it  can  either  reduce  instructional  time  to  reach  instructional 
goals  by  about  one-third,  or  increase  the  skills  and  knowledge  acquired  by  about  one- 
third  while  holding  instructional  time  constant. 

The  long-term  vision  for  ADL  is  an  extrapolation  from  such  developments  as 
portable,  increasingly  accessible  computing,  the  global  information  infrastructure, 
modular  object-oriented  architectures,  and  natural  language  processing.  The  march 
toward  devices  that  might  be  described  as  personal  learning  associates  seems  inevitable. 
These  devices  will  act  as  personal  accessories.  They  will  respond  on  demand  to  requests 
for  education,  training,  and  performance  aiding  by  assembling  relevant  objects  from  the 
global  infrastructure  and  engaging  the  user  in  guided  conversations  to  enhance  the 
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knowledge  and  skills  and/or  problem  solving  capabilities  of  individuals  and/or  groups  of 
dispersed  individuals  whose  devices  are  wirelessly  linked  together. 
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Research  Foundations  for  the 
Advanced  Distributed  Learning  (ADL)  Initiative 


A.  The  Requirement 

About  1.1  million  U.S.  forces  are  dispersed  across  the  Continental  United  States  and 
an  additional  1.4  million  forces  are  spread  across  50  foreign  countries.  Individuals  in 
these  forces  need  to  be  capable  of  independent  thought  and  action.  They  also  need  to  be 
able  to  continue  their  training  and  career  growth  when  they  are  far  removed  from  military 
schoolhouses,  expert  mentors,  or  others  in  their  own  career  specialties. 

Additional  complications  arise  from  the  rapid  growth  in  the  amount  and  complexity 
of  information  that  individuals  at  all  levels  must  integrate  and  prioritize.  The  trend  has 
produced  an  increasing  demand  for  what  Wulfeck  and  Wetzel-Smith  (2008)  and  Wetzel- 
Smith  and  Wulfeck  (2010)  have  described  as  “incredibly  complex  tasks.”  Today,  about 
15%  of  military  tasks  are  abstract,  multidimensional,  non-linear,  dynamic,  and  inter¬ 
dependent.  The  dynamic  nature  of  these  tasks  and  the  evolving  operational  environment 
require  that  individuals  receive  up-to-date  training  and  performance  assistance. 

Finally,  the  difficulty  of  all  military  tasks  is  exacerbated  by  the  dispersal  of  units 
and  the  individuals  serving  within  these  units.  Human  decisions  and  actions  must  be 
coordinated  within  and  across  teams  whose  members  may  be  globally  dispersed  in  other 
nations,  immersed  in  other  cultures,  and  serving  at  widely  varied  command  levels.  The 
ability  to  communicate,  coordinate,  and  perform  tasks  under  these  conditions  may  not 
guarantee  success,  but  the  consequences  of  its  absence  are  severe. 

These  operational  conditions  have  created  an  imperative  to  ensure  that  “learning” — 
education,  training,  and  task,  job,  and  decision  aiding — is  rapidly  available  on  demand, 
anytime  and  anywhere,  to  uniformed  and  civilian  individuals  and  teams  at  all  levels  of 
responsibility. 

B.  The  Technical  Opportunity:  A  Third  Revolution  in  Learning 

In  response  to  this  requirement,  ADL  is  riding  and  contributing  to  a  third  revolutio¬ 
nary  wave  in  learning.  The  first  revolution  in  learning  occurred  about  5,000  years  ago, 
with  the  invention  of  writing — the  use  of  graphic  tokens  to  represent  syllables  of  sound. 
Before  the  invention  of  writing,  learning  activity  appears  to  have  been  conducted  as  a 
tutorial  conversation  between  a  learner  and  a  sage,  or  at  least  someone  who  had  the 
knowledge  and  skill  the  learner  sought  to  acquire.  With  writing,  a  learner  could  access 
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knowledge  and  skill  without  this  face-to-face  interaction.  Writing  allowed  the  content  of 
ideas  and  instruction  to  transcend  time  and  place. 

The  second  revolution  in  learning  occurred  with  the  invention  of  books  printed  from 
moveable  type — first  in  China  around  1000  A.D.  and  then  in  Europe  in  the  mid- 1400s 
(Kilgour,  1998).  With  books,  the  dissemination  of  knowledge  and  skills  through  writing 
became  scalable.  Once  content  was  produced,  it  could  be  made  widely  available  and 
became  increasingly  inexpensive  as  printing  technology  developed.  However,  with 
writing  and  printing,  the  dissemination  of  content  was  passive.  It  lacked  the  tutorial 
interactivity  that  had  been  the  foundation  of  learning  for  the  previous  100,000  years  or  so 
of  human  existence. 

Enter  computer  technology,  with  its  ability  to  adapt  rapidly,  in  real  time,  to  the 
changing  demands,  needs,  and  circumstances  of  learners  and  learning.  Computer  technol¬ 
ogy  allows  not  just  content,  but  also  instructional  strategies,  techniques,  and  interactions 
to  become  inexpensively  ubiquitous  and  available  on  demand — anytime,  anywhere.  It 
may  be  fomenting  a  third  revolution  in  learning.  ADL  is  a  response  and  a  contributor  to 
this  third  revolutionary  possibility. 

“Learning”  in  ADL  is  used  as  a  catch-all  designator  for  education,  training,  and 
performance/decision  aiding.  “Distributed”  in  ADL  is  not  just  another  word  for  distance. 
It  signifies  learning  that  can  be  provided  in  classrooms  with  a  teacher  present,  in  the  field 
linking  together  widely  dispersed  instructors  and  students,  or  standing  alone  with  no 
instructor  other  than  the  computer  itself.  “Advanced”  in  ADL  implies  affordable, 
interactive,  adaptive,  on-demand  instruction  using  computer  technology  so  that  it  can  be 
delivered  anytime,  anywhere. 

The  ADL  purpose,  from  its  inception,  has  been  to  ensure  access  to  the  highest  qual¬ 
ity  education,  training,  and  performance/decision  aiding  tailored  to  individual  needs  and 
delivered  cost  effectively  and  any  time/any  where. 

C.  Evidence:  Research  and  Development  (R&D)  Foundations 

What  evidence  suggests  that  computer  technology  might  be  effecting  this  third  revo¬ 
lution?  What  have  we  learned  from  research  on  computer  uses  in  instruction?  Some  key 
findings  can  be  summarized  as  follows: 

•  Although  individualized  learning  tailored  to  the  needs  of  individual  students  has 
long  been  viewed  as  an  imperative,  it  has  also  been  viewed  as  unaffordable 
(Scriven,  1975).  With  few  exceptions,  we  cannot  afford  one  instructor  for  every 
student — an  Aristotle  for  every  Alexander.  Computer  technology  can  make  this 
imperative  affordable.  A  core  argument  for  ADL,  then,  is  not  for  technology  but 
for  making  individualization  affordable. 
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•  The  instructional  technologies  targeted  by  ADL  have  been  found  to  be  more 
effective  than  typical  classroom  instruction  across  many  instructional  objectives 
and  subject  matters. 

•  ADL  is  generally  less  costly,  offering  greater  return  on  investment  (ROI)  than 
current  instructional  approaches,  especially  when  many  widely  dispersed 
students  must  be  served. 

•  ADL  allows  education,  training,  and  performance/decision  aiding  and  problem 
solving  to  be  delivered  from  the  same  knowledge  bases  on  platforms  ranging 
from  hand-held  devices,  to  large  desktop  computers,  to  capabilities  embedded  in 
operational  equipment. 

These  arguments  have  been  made  for  the  computer-assisted  approaches  used  by 
ADL  for  the  last  40-50  years  (e.g.,  Alpert  and  Bitzer  (1970),  Atkinson  (1968),  Coulson 
(1962),  Galanter  (1959),  and  Suppes  (1966)).  They  have  been  repeatedly  validated  by 
empirical  research  and  practical  experience.  Statistical  findings  from  this  work  have  been 
summarized  by  a  “Rule  of  Thirds”  (Fletcher,  1997,  2004).  This  rule  states  that  application 
of  the  technologies  on  which  ADL  is  based  reduces  the  cost  of  instruction  by  about  one- 
third.  In  addition,  the  application  of  these  technologies  can  either  reduce  instructional 
time  to  reach  instructional  goals  by  about  one-third  or  increase  the  skills  and  knowledge 
acquired  by  about  one-third  while  holding  instructional  time  constant. 

The  following  sections  discuss  more  specifically  the  R&D  behind  these  arguments 
and  the  Rule  of  Thirds. 

D.  Individualization:  Tutorial  Instruction 

The  argument  for  ADL  technology  begins  with  an  issue  that  arises  independently 
from  applications  of  technology.  It  concerns  the  effectiveness  of  classroom  instruction, 
involving  one  instructor  for  20-30  (or  more)  students,  compared  to  individual  tutoring, 
involving  one  instructor  for  each  student.  Empirical  results  from  comparisons  of  this  sort 
are  shown  in  Figure  1,  adapted  from  Bloom  (1984). 

Bloom  combined  findings  from  three  empirical  studies  that  compared  tutoring  with 
one-on-many  classroom  instruction.  The  result  of  such  comparisons  showed  the  tutored 
students  to  have  learned  more,  and  the  result  is  not  surprising.  The  surprise  is  the  size  of 
the  difference.  Overall,  as  Figure  1  suggests,  the  difference  was  found  to  be  two  standard 
deviations.  It  suggests  that  one-on-one  tutoring,  with  instructional  time  held  constant,  can 
raise  the  performance  of  mid-level  50th  percentile  students  roughly  to  that  of  98th 
percentile  students.  These  and  similar  empirical  findings  suggest  that  differences  between 
one-on-one  tutoring  and  typical  classroom  instruction  are  not  only  likely  but  very  large. 
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Achievement 

Figure  1.  Individual  Tutoring  Compared  to  Classroom  Instruction 


Most  importantly  for  training  applications,  the  shapes  of  these  distributions  support 
Como  and  Snow’s  (1986)  suggestion  that  the  individualization  provided  by  tutorial 
instruction  helps  guarantee  that  all  learners  reach  some  basic  level  of  competency. 

What  accounts  for  the  success  of  one-on-one  tutoring?  The  research  summarized 
below  suggests  that  it  is  primarily  due  to  (1)  the  ability  of  tutors  and  their  students  to 
engage  in  many  more  interactions  per  unit  of  time  than  is  possible  in  a  classroom  and 
(2)  the  ability  of  tutors  to  tailor  pace,  sequencing,  and  content  to  the  needs,  capabilities, 
goals,  interests,  and  values  of  individual  students. 

1.  Interactivity 

With  regard  to  the  first  tutorial  capability  (the  intensity  of  instructional  interaction), 
Graesser  and  Person  (1994)  reported  the  following: 

•  Average  number  of  questions  by  a  teacher  of  a  class  in  a  classroom  hour:  3 

•  Average  number  of  questions  asked  by  a  tutor  and  answered  by  a  student  during 
a  tutorial  hour:  120-145 

•  Average  number  of  questions  asked  by  any  one  student  during  a  classroom  hour: 

0.11 

•  Average  number  of  questions  asked  by  a  student  and  answered  by  a  tutor  during 
a  tutorial  hour:  20-30. 

These  data  show  great  differences  in  interactivity  and  intensity  between  tutorial  and 
classroom  instruction.  This  level  of  interactivity,  by  itself,  may  account  for  a  substantial 
portion  of  the  success  of  tutorial  over  classroom  instruction. 

Is  this  level  of  interactivity  found  in  instruction  using  ADL  technology?  Early 
studies  of  computer-assisted  instruction  in  reading  and  arithmetic  found  that  students  in 
grades  K-6  were  answering  8-10  individually  selected  and  assessed  questions  each 
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minute  (Fletcher  and  Atkinson,  1972;  Fletcher  and  Suppes,  1972).  This  level  of 
interactivity  extrapolates  to  480-600  such  questions  an  hour  if  students  were  to  sustain 
this  level  of  interaction  for  60  minutes. 

2.  Tailoring  Pace,  Sequencing,  and  Content 

With  regard  to  the  second  tutorial  capability  (tailoring  the  session  content),  tutors 
adjust  the  content  and  sequence  of  instruction  to  the  needs  of  their  students.  All  these 
adjustments  relate  to  pace — the  rate  or  speed  with  which  students  are  allowed  to  proceed 
through  instructional  material. 

Many  classroom  instructors  have  been  struck  by  the  differences  in  the  pace  with 
which  their  students  learn.  Their  observations  are  confirmed  by  research.  For  instance, 
consider  some  findings  on  the  time  it  takes  for  different  students  to  reach  the  same 
instructional  objectives: 

•  Ratio  of  time  needed  by  fastest  and  slowest  students  to  reach  mathematics 
objectives:  4  to  1  (Suppes,  Fletcher,  andZanotti,  1975;  1976) 

•  Overall  ratio  of  time  needed  by  fastest  10%  and  slowest  10%  of  K-8  students  to 
reach  objectives  in  a  variety  of  subjects:  5  to  1  (Gettinger,  1984) 

•  Ratio  of  time  needed  by  fastest  and  slowest  undergraduates  in  a  major  research 
university  to  learn  a  programming  language:  7  to  1  (Private  communication, 
Corbett,  1998). 

The  differences  in  the  speed  with  which  students  learn  are  not  surprising,  but,  as 
with  tutoring,  the  magnitudes  of  the  differences  are  surprising.  Although  the  speed  with 
which  different  students  reach  instructional  objectives  is  not  independent  of  ability, 
research  has  found  that  it  is  most  directly  keyed  to  prior  knowledge  (Dochy,  Segers,  and 
Buehl,  1999;  Tobias,  1989).  Students  in  military  education  and  training  bring  with  them  a 
wide  variety  of  backgrounds  and  life  experiences — often  much  wider  than  those  found 
among  K-12  students.  Adjusting  the  pace  of  instruction  to  their  individual  needs  may  be 
especially  important  for  them. 

The  challenge  this  diversity  presents  to  classroom  instructors  is  daunting.  Typically, 
the  instructors  focus  on  some  of  their  students  and  leave  the  others  to  fend  for  them¬ 
selves.  This  pattern  is  especially  true  in  training  settings  where  the  primary  task  is  to 
enable  as  many  learners  as  possible  to  cross  a  specific  threshold  of  knowledge  and  skill. 
Technology  alleviates  this  difficulty  because  it  allows  each  learner  to  proceed  as  rapidly 
or  as  slowly  as  needed.  Learners  can  skip  what  they  already  know  and  concentrate  on 
what  they  need  to  learn. 

The  degree  to  which  individualization  of  sequence  and  content  matters  is  to  some 
extent  addressed  by  studies  comparing  individualized  branching  with  fixed-content, 
linear  sequencing.  Two  of  the  early  studies  were  performed  by  Fowler  (1980)  and  Verano 
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(1987).  Both  of  these  researchers  used  computer-controlled  videodisc  instruction  in  their 
experiments.  Fowler  compared  branched  presentations  with  linear  instruction  in  which 
precisely  the  same  materials  were  held  to  a  fixed-content,  linear  sequence.  She  reported 
an  effect  size  of  0.72  (roughly,  an  improvement  from  the  50th  to  76th  percentile)  in  the 
ability  to  operate  and  locate  faults  on  a  movie  projector.  Verano  also  compared  an 
interactive,  adaptive,  branching  approach  with  a  strictly  linear  approach  for  presenting 
instructional  material  in  beginning  Spanish.  He  reported  an  effect  size  of  2.16  (roughly, 
an  improvement  from  the  50th  to  98th  percentile)  in  end-of-course  knowledge.  These  two 
studies,  among  others,  suggest  that  individualization  of  sequence  and  content  matters — 
perhaps  a  great  deal. 

E.  Technology- Assisted  Learning 

1.  Time  Savings 

One  of  the  most  stable  findings  in  the  comparisons  of  technology-based  instruction 
and  conventional  instruction  (which  uses  lecture,  text,  and  experience  with  equipment 
(e.g.,  in  the  laboratory))  concerns  instruction  time  savings.  Table  1  presents  these 
findings. 


Table  1.  Percent  Time  Savings  for  Technology-Based  Instruction 


Study  (Reference) 

Number  of 
Studies 
Reviewed 

Average 

Time  Saved 
(Percent) 

Orlansky  and  String  (1977)  (Military  Training) 

13 

54 

Fletcher  (1991)  (Higher  Education) 

8 

31 

Kulik  (1994)  (Higher  Education) 

17 

34 

Kulik  (1994)  (Adult  Education) 

15 

24 

As  the  table  shows,  Orlansky  and  String  (1977)  reported  that  reductions  in  time  to 
reach  instructional  objectives  averaged  about  54%  in  13  assessments  of  technology-based 
military  training.  Fletcher  (1991)  reported  an  average  time  reduction  of  31%  in  8  assess¬ 
ments  of  IMI  applied  in  higher  education.  Kulik  reported  average  time  reductions  of  34% 
in  17  assessments  of  technology  used  in  higher  education  and  24%  in  15  assessments  of 
adult  education  (Kulik,  1994).  Each  of  these  reviews  covered  different  sets  of  evaluation 
studies  that  compared  technology-based  instruction  and  conventional  classroom  instruc¬ 
tion  involving  lecture,  texts,  and  perhaps  laboratory  examples.  Overall,  it  seems 
reasonable  to  expect  that  technology-based  instruction  will  reduce  by  about  30%  the  time 
students  take  to  reach  a  variety  of  objectives. 
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2.  Costs 


An  example  of  the  cost  benefits  of  this  reduction  in  time  to  learn  can  be  seen  in 
residential,  specialized  skill  training.  The  DoD  spends  about  $6.5  billion  a  year  on  this 
training,  which  is  the  “schoolhouse”  training  individuals  receive  after  Basic,  or  initial 
accession,  training.  It  qualifies  individuals  for  the  many  technical  jobs  (e.g.,  wheeled 
vehicle  mechanics,  radar  operators,  avionics  technicians,  medical  technicians,  and  so 
forth)  needed  to  perform  military  operations.  It  does  not  include  the  costs  of  aircraft  pilot 
training,  officer  education,  or  training  provided  in  military  units. 


Extrapolated  from  an  earlier  analysis  by  Angier,  Fletcher,  and  Horowitz  (1991), 
Figure  2  shows  the  annual  reductions  in  costs  that  would  result  if  instruction  time  were 
reduced  by  30%  for  20,  40,  60,  and  80%  of  military  personnel  who  complete  residential, 
specialized  skill  training  each  year.  For  instance,  if  the  DoD  reduced  by  30%  the  time  to 
train  20%  of  the  personnel  undergoing  specialized  skill  training,  it  would  save  about 
$428  million  per  year.  If  it  were  to  do  so  for  60%  of  these  personnel,  it  would  save  about 
$1,284  million  per  year — or  about  20%  of  the  funds  allocated  for  specialized  skill 
training.  Specialized  skill  training  is  particularly  amenable  to  the  use  of  ADL  technolo¬ 
gies.  Use  of  these  technologies  by  60%  of  specialized  skill  trainees  is  not  an  unreasonable 
expectation. 
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Figure  2.  Cost  Savings  ($  2008)  in  Specialized  Skill  Training 
Assuming  a  30%  Reduction  in  Training  Time 


Saving  30%  of  training  time  may  be  a  conservative  target.  Commercial  enterprises 
that  develop  technology-based  instruction  for  the  DoD  regularly  base  their  bids  on  the 
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expectation  that  they  can  reduce  instructional  time  by  50%.  Noja  (1987)  has  reported 
time  savings  as  high  as  80%  with  the  use  of  technology-based  instruction  in  training 
operators  and  maintenance  technicians  for  the  Italian  Air  Force. 

Two  other  sources  of  cost  savings  with  ADL  technologies  are  not  considered  in  the 
Figure  2  data.  First,  the  cost  models  used  to  generate  the  data  in  Figure  2  assume 
reductions  in  training  time  in  residential  settings  but  do  not  take  into  account  using  ADL 
technologies  to  distribute  some  of  that  training  to  operational  units — thereby  reducing 
change  of  station  or  temporary  duty  costs.  Second,  ADL  technologies  can  be  used  to 
simulate  expensive  equipment,  operational  environments,  and  interpersonal  situations — 
thereby  not  just  reducing  costs,  but  also  increasing  safety,  enhancing  visualization,  and 
allowing  time  to  be  sped  up  or  slowed  down  as  needed  for  the  training. 

Perhaps  more  importantly  for  military  applications,  ADL  technologies  can  prepare 
individuals  more  quickly  for  operational  duty.  In  this  way,  these  technologies  act  as  force 
multipliers  by  increasing  readiness  and  operational  effectiveness  without  increasing  per¬ 
sonnel  costs. 


a.  Instructional  Effectiveness 

Research  data  suggest  that  savings  from  using  ADL  technologies  do  not  come  at  the 
expense  of  instructional  effectiveness.  Empirical  findings  report  the  opposite.  Figure  3 
shows  effect  sizes  from  several  reviews  of  studies  that  compared  conventional  instruction 
with  technology-based  instruction. 
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Figure  3.  Some  Effect  Sizes  for  Studies  Comparing 
Technology-Based  Instruction  With  More  Conventional  Approaches 
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In  Figure  3,  computer-based  instruction  (CBI)  summarizes  the  results  from 
233  studies  that  involved  a  straightforward  application  of  computer  presentations  using 
text,  graphics,  and  some  animation,  as  well  as  some  degree  of  individualized  interaction. 
The  effect  size  of  0.39  standard  deviations  suggests,  roughly,  an  improvement  of  50th 
percentile  students  to  the  performance  levels  of  65th  percentile  students. 

Interactive  multimedia  instruction  (IMI)  involves  more  elaborate  interactions  adding 
audio,  animation,  and  video  and  generally  taking  advantage  of  the  multimedia  effect 
(Fletcher  and  Tobias,  2005;  Mayer,  2005).  These  added  capabilities  evidently  increase 
achievement.  They  show  an  average  effect  size  of  0.50,  which  suggests  an  improvement 
of  50th  percentile  students  to  the  performance  levels  of  69th  percentile  students. 

Intelligent  tutoring  systems  (ITSs)  involve  a  capability  that  has  been  developing 
since  the  late  1960s  (e.g.,  Carbonell,  1970;  Sleeman  and  Brown,  1982).  In  this  approach, 
an  attempt  is  made  to  directly  mimic  the  one-on-one  dialogue  that  occurs  in  tutorial  inter¬ 
actions.  A  key  goal  of  these  systems  is  to  generate  computer  presentations  and  responses 
in  real  time  and  on  demand  as  needed  or  requested  by  learners.  Instructional  designers  do 
not  need  to  anticipate  and  pre-store  them.  This  approach  is  computationally  more  sophis¬ 
ticated  and  more  expensive  to  produce  than  standard  computer-based  instruction. 
However,  its  cost  can  be  justified  by  the  increase  in  average  effect  size  to  0.84  standard 
deviations,  which  suggests,  roughly,  an  improvement  of  50th  percentile  students  to  the 
performance  levels  of  80th  percentile  students.  As  will  be  discussed  later,  ROI  is  much 
more  sensitive  to  scaling  and  delivery  costs  of  instruction  than  to  the  initial  costs  to 
design  and  develop  it. 

A  selected  group  of  ITSs  (“Recent  Intelligent  Tutors”)  was  considered  just  to  see 
how  far  these  systems  are  progressing.  The  average  effect  size  of  1.05  standard 
deviations  for  these  applications  is  promising.  It  represents,  roughly,  an  improvement  of 
50th  percentile  students  to  the  performance  levels  of  85th  percentile  students. 

The  extensive  tailoring  of  instruction  that  generative,  ITSs  provide  to  meet  the 
needs  of  individual  students  can  only  be  expected  to  increase.  Such  systems  may  raise  the 
bar — well  past  Bloom’s  2-Sigma  challenge — for  the  ultimate  effectiveness  of  ADL-based 
instruction. 

b.  Student  Attitudes 

Student  attitudes  toward  instruction  can  affect  its  effectiveness  and  efficiency.  Many 
evaluations  of  technology-based  instruction  simply  ask  students  if  they  prefer  it  to  more 
conventional  classroom  approaches.  Greiner  (1991)  reviewed  these  evaluations  and  found 
overall  that  70-80%  of  students  who  were  polled  preferred  technology-based  approaches 
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to  other  approaches.  When  students  reported  that  they  did  not  prefer  the  technology- 
based  approaches,  the  reasons  were  usually  because  of  implementation  or  technical 
problems  with  the  technology  and  not  the  instructional  approach  itself. 

McKinnon,  Nolan,  and  Sinclair  (2000)  completed  a  thorough  3-year  study  of 
student  attitudes  toward  the  use  of  technology-based  learning  and  productivity  tools  such 
as  spreadsheets,  databases,  graphics,  desktop  publishing,  and  statistical  processing.  The 
attitudes  of  the  students  toward  technology  use  slackened  as  the  novelty  of  using  the 
technology  wore  off.  However,  their  attitudes  remained  positive  and  significantly  more 
positive  than  those  of  students  who  did  not  have  access  to  the  technology  throughout  the 
3  years  of  the  study. 

c.  ROI 

Knowing  that  we  can  use  ADL  technologies  to  reduce  learning  time,  particularly 
time  to  learn  journeyman  skills  such  as  remembering,  understanding,  and  applying  facts, 
simple  concepts,  and  straight-forward  procedures,  what  might  an  investment  in  these 
technologies  return? 

One  way  to  answer  this  question  is  by  applying  the  findings  presented  earlier  in 
Figure  2,  which  only  considered  savings.  Using  the  analysis  underlying  that  figure,  we 
can  wrap  in  both  the  savings  and  the  costs  to  achieve  them  using  an  ROI  model.  This 
model  simply  reduces  to  the  ratio  of  the  net  return  (savings  in  this  case)  to  the  costs  as 
shown  in  the  following 


Savings  -  Costs 
Costs 

We  can  begin  by  assuming  (conservatively)  a  30%  reduction  in  training  time 
achieved  through  the  use  of  ADL  technologies  by  (conservatively)  40%  of  residential 
specialized  training  students.  From  Figure  2  and  the  analysis  on  which  it  is  based 
(Angier,  Fletcher,  and  Horowitz,  1991),  these  assumptions  suggest  annual  savings  of 
$854  million.  Given  this  result,  the  next  step  is  to  determine  the  costs  to  design,  develop, 
and  deliver  ADL  instruction  to  40%  of  residential  specialized  skill  training  students. 

Using  ADL  technology,  how  much  would  it  cost  to  render  the  training  needed  by 
40%  of  specialized  skill  students?  According  to  the  last  published  Military  Manpower 
Training  Report  (2002),  the  average  Specialized  Training  course  length  across  all  four 
Services  was  about  57  training  days.1  If  that  number  were  reduced  by  30%,  the  training 


1  The  2002  Military  Manpower  Training  Report  is  the  last  such  report  issued  by  the  OUSD(P&R). 
Changes  in  specialized  skill  training  since  2002  are  assumed  to  have  been  small.  This  present  analysis 
uses  training  and  manpower  data  from  the  2002  report  but  uses  2008  cost  data  for  personnel  and  ADL 
course  development  and  delivery. 
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course  length  would  amount  to  about  40  days.  Military  personnel  in  training  billets  may 
be  required  to  perform  additional  duties,  but,  assuming  8  hours  a  day  in  course  training 
amounts  to  about  320  hours  of  training.  About  357,700  officers  and  enlisted  personnel 
completed  Specialized  Training  in  FY  2002.  Forty  percent  of  that  number  amounts  to 
143,080  learners.  In  effect  then,  320  hours  of  ADL  training  would  have  to  be  produced 
and  then  delivered  to  143,080  learners. 

Estimates  to  produce  an  hour  of  computer-assisted  training  vary  widely  and  depend 
on  the  content,  instructional  strategy,  and  pay  and  allowances  for  subject  matter  experts, 
“authors,”  and  computer  programmer/analysts.  One  source  of  estimates  comes  from  the 
Joint  Knowledge  Development  and  Distribution  Capability  (JKDDC),  which  is  producing 
Web-based  individual  training  programs  for  joint  assignments  and  operations.  As  of  May 
2008,  JKDDC  had  produced  over  200  courses,  with  more  than  65,000  course  completions 
(Camacho,  2009).  In  the  first  quarter  of  FY  2008,  the  costs  for  JKDDC  to  develop  an 
hour  of  instruction  were  about  $14,000,  and  the  costs  to  deliver  it  were  about  $4. 

We  might  then  assume  that  it  would  cost  about  $4.48  million  to  produce  320  hours 
of  ADL  specialized  skill  training  and  an  additional  $183.14  million  to  deliver  it  to 
143,080  students.  The  investment  for  the  first  year  of  this  training  would  cost  about 
$187.62  million. 

Developing  and  delivering  technology-based  instruction  to  40%  of  Specialized  Skill 
students  (about  143,080  students)  would  require  30%  fewer  hours  to  complete  their 
training — a  savings  of  about  136  hours  for  each  student.  Assuming  a  composite  of  $42 
per  hour  in  pay  and  allowances  (Military  Personnel  Composite  Standard  Pay,  2009),  this 
use  of  technology-based  instruction  would  amount  to  (42  x  143,080  x  136)  =  $817.27 
million.  Plugging  these  data  and  assumptions  into  an  ROI  calculation  yields 

$8 1 1 21 M  -  $  1 87.62M  .  . . 

- =  3.36 

S187.62M 

Under  these  assumptions,  an  investment  in  ADL  technology  will  return  about  $3.36 
for  every  dollar  invested. 

After  the  first  year,  the  costs  to  develop  the  instruction  would  be  reduced  to  what¬ 
ever  is  required  to  maintain  and  update  the  course,  but  the  ROI  is  not  particularly 
sensitive  to  development  costs.  It  seems  far  more  sensitive  to  delivery  costs.  They  are 
included  in  this  analysis  for  the  delivery  of  the  ADL  instruction  but  not  for  the  classroom 
instruction  it  replaces.  If  the  savings  in  training  delivery  costs  were  fully  considered  in 
this  calculation,  the  ROI  would  increase  substantially.  Also,  as  suggested  earlier,  30% 
time  savings  is  likely  to  be  an  underestimate  of  the  student  time  in  training  that  can  be 
saved.  Further,  even  though  this  analysis  assumes  Web-based  capability  for  delivering 
the  instruction,  it  does  not  take  into  account  reductions  in  travel  and  temporary  duty 
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costs,  which,  if  included,  would  further  increase  ROI.  Finally,  the  administrative  efficien¬ 
cies,  improved  tracking  and  assurance  of  student  progress,  and  other  benefits  provided  by 
ADL  technology — benefits  that  were  not  considered  in  the  previous  analysis — have  their 
place  and  would  continue  to  argue  for  its  use. 

Less  time  in  school  means  more  time  on  the  job.  Savings  in  time  needed  to  reach 
training  objectives  not  only  reduce  training  costs,  but  also  increase  the  supply  of  people 
for  operational  forces  without  increasing  the  number  of  people  in  uniform.  Ways  to 
account  for  accompanying  increases  in  readiness  and  effectiveness  due  to  force 
multiplication  remain  to  be  determined,  but  they,  rather  than  savings  in  training  costs, 
may  be  the  most  significant  impact  of  reducing  the  time  required  by  operational  forces  to 
reach  performance  levels. 

In  short,  a  significant  return  seems  likely  to  result  from  an  investment  to  convert 
some  proportion  of  training  to  ADL  technology.  This  value  arises  primarily  from  the 
reduction  of  student  time  spent  in  the  training  infrastructure.  Even  with  ADL,  seasoned 
military  personnel  must  provide  additional  training,  mentoring,  and  monitoring  for  people 
early  in  their  careers.  Certainly,  efforts  must  be  made  to  preserve  the  camaraderie  and 
espirit  de  corps  gained  by  students  undergoing  the  rigors  of  training  together.  The  argu¬ 
ment  for  ADL  is  not  to  suggest  the  massive  replacement  of  people  by  the  technology.  It 
is,  instead,  that  ADL  reduces  costs  to  train  and  increases  force  effectiveness  by  releasing 
people  sooner  from  the  training  infrastructure  and  ensuring  their  competencies.  A  cost- 
effective,  optimal  balance  between  the  use  of  ADL  and  more  people-intensive  approaches 
to  training  and  education  remains  to  be  determined. 

3.  Performance  Aiding 

Most  of  the  discussion  to  this  point  has  focused  on  education  and  training  applica¬ 
tions  using  ADL  technology.  Something  remains  to  be  said  about  its  use  in  providing  on- 
demand  performance  aiding.  The  term  Interactive  Electronic  Technical  Manual  (IETM) 
is  a  generic  label  for  such  a  device  (Gafford  &  Heller,  in  press).  Fletcher  and  Johnston 
(2002)  presented  data  on  effectiveness  and  costs  of  several  portable,  electronic  mainten¬ 
ance  performance  systems,  one  of  which  was  the  Integrated  Maintenance  Information 
System  (IMIS). 

IMIS  was  a  wearable,  computer-based  performance  aid  for  avionics  maintenance. 
Thomas  (1995)  compared  the  performance  of  12  Avionics  Specialists  and  12  Airplane 
General  (APG)  Technicians  in  troubleshooting  three  F-16  avionics  subsystems.  Within 
each  of  the  two  groups  of  subjects,  six  of  the  fault  isolation  problems  were  performed 
using  paper-based  technical  orders  (TOs)  (Air  Force  technical  manuals),  and  six  were 
performed  using  IMIS. 
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Training  for  APG  Technicians  includes  all  aspects  of  aircraft  maintenance,  only  a 
small  portion  of  which  concerns  avionics.  In  contrast,  Avionics  Specialists  must  meet 
higher  selection  standards  and  receive  16  weeks  of  specialized  training  that  focuses  on 
avionics  maintenance.  Costs  to  train  APG  Technicians  are  about  half  the  costs  for 
Avionics  Specialists.  Table  2  shows  the  results  of  the  study. 

Table  2.  Maintenance  Performance  of 

12  Air  Force  Avionics  Specialists  and  12  APG  Technicians  Using  TOs  and  IMIS 


Correct  Time  to  Average  Time  to 

Solutions  Solution  Number  of  Order  Parts 

Technicians/Performers  (Percent)  (Minutes)  Parts  Used  (Minutes) 


TOs 

IMIS 

TOs 

IMIS 

TOs 

IMIS 

TOs 

IMIS 

Avionics  Specialists 

81.9 

100.0 

149.3 

23.6 

8.7 

6.4 

19.4 

1.2 

APG  Technicians 

69.4 

98.6 

175.8 

124.0 

8.3 

5.3 

25.3 

1.5 

As  shown  in  the  table,  findings  of  the  study  were  as  follows: 

•  Avionics  Specialists  using  TOs  compared  with  those  using  IMIS.  The 

Avionics  Specialists  using  IMIS  found  more  correct  solutions  in  less  time,  used 
fewer  parts  to  do  so,  and  took  less  time  to  order  the  parts.  All  these  results  were 
statistically  significant.  The  results  concerning  time  to  order  parts  are  to  be 
expected  because  IMIS  automated  much  of  this  process. 

•  APG  Technicians  using  TOs  compared  with  those  using  IMIS.  The  APG 
Technicians  using  IMIS  found  more  correct  solutions  in  less  time,  used  fewer 
parts  to  do  so,  and  took  less  time  to  order  the  parts.  All  these  results  were 
statistically  significant. 

•  APG  Technicians  using  IMIS  compared  with  Avionics  Specialists  using 
TOs.  The  APG  Technicians  using  IMIS  found  more  correct  solutions  in  less 
time,  used  fewer  parts  to  do  so,  and  took  less  time  to  order  the  parts  than  did 
Avionics  Specialists  using  paper-based  TOs.  All  these  results  were  statistically 
significant. 

•  APG  Technicians  using  IMIS  compared  with  Avionics  Specialists  using 
IMIS.  The  APG  Technicians  performed  just  about  as  well  as  the  Avionics 
Specialists  and  even  slightly  better  in  the  number  of  parts  used.  None  of  these 
comparisons  were  statistically  significant. 

The  economic  promise  suggested  by  these  results  could  well  vanish  if  the  costs  to 
provide  the  performance  aid  (i.e.,  the  IMIS)  exceed  the  costs  they  otherwise  save. 
Teitelbaum  and  Orlansky  (1996)  estimated  IMIS  reductions  in  depot-level  maintenance, 
organizational-level  maintenance,  and  maintenance  and  transportation  of  inventories  of 
spare  parts.  They  found  annual  savings  from  the  use  of  IMIS  at  about  $38  million  for  the 
full  Air  Force  fleet  of  F- 16s.  They  assumed  an  8-year  useful  life  for  IMIS  and  estimated 
about  $18  million  per  year  to  maintain  and  update  it  and  amortize  its  development  costs. 
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The  result  is  a  benefit  of  about  $20  million  per  year  in  net  savings  or  an  ROI  of  about 
1.11,  which  excludes  the  significant  impact  of  IMIS  on  sortie  rate,  readiness,  and 
operational  effectiveness. 

F.  The  ADL  Vision 

The  long-term  ADL  vision  is  an  extrapolation  from  such  developments  as  portable, 
increasingly  accessible  computing  (including  hand-held,  worn,  or  even  implanted  com¬ 
puters),  the  global  information  infrastructure  (currently  manifest  in  the  World  Wide  Web 
with  its  multifarious  search  engines),  modular  object-oriented  architectures,  Web  2.0 
technologies,  and  natural  language  processing.  The  march  toward  devices  that  might  be 
described  as  personal  learning  associates  seems  inevitable. 

As  currently  envisioned,  these  devices  will  act  as  personal  accessories.  They  will 
respond  on  demand  to  each  individual’s  needs  for  education,  training,  and  performance 
aiding  by  assembling  relevant  objects  from  the  global  infrastructure  and  engaging  the 
user  in  guided  conversations,  such  as  those  described  by  Hu,  Graesser,  and  Fowler  (in 
press),  to  enhance  user  knowledge  and  skills  and/or  problem  solving  capabilities. 
Learning  in  these  cases  is  not  a  matter  of  just  working  through  pre-specified  lessons  but 
is  a  return  to  the  100,000  year  old  tutorial  practice  of  an  individual  and  a  sage  working 
together  to  enhance  knowledge  and  skill.  In  this  case,  the  human  sage  is  supplanted  by  a 
computational  device  that  has  access  to  something  approaching  the  whole  of  human 
knowledge  carried  throughout  the  global  information  infrastructure. 

As  described  Dodds  and  Fletcher  (2004)  and  by  Wisher  (in  press),  objects  drawn 
from  the  global  infrastructure  must  be  portable,  durable,  reusable,  and  accessible. 
Gallagher  (in  press)  discusses  the  history  and  development  of  the  Sharable  Content 
Object  Reference  Model  (SCORM),  which  can  ensure  that  objects  have  the  first  three  of 
these  qualities. 

In  addition,  the  objects  must  be  accessible.  As  discussed  in  more  detail  by  Lannom 
(in  press),  the  development  of  the  Content  Object  Repository  Registration/Resolution 
Architecture  (CORDRA)  and  its  use  by  the  ADL  registry  provide  global  visibility  for 
objects  while  allowing  their  developers  to  retain  control  over  access  to  them.  They  are 
then  available  for  reuse  or  repurposing. 

These  developments  have  provided  ways  for  objects  to  be  identified  and  collected 
for  local  use  from  the  global  information  infrastructure.  The  continued  operation  of 
Moore’s  Law  and  the  market-driven  effort  to  imbue  computer  technology  with  natural 
language  understanding  should  ensure  development  of  affordable,  mobile,  conversation- 
capable  computing.  The  major  issue  that  remains  for  ADL  is  development  of  the 
envisioned  individualized  tutorial  capabilities.  Progress  and  promise  can  be  seen  in  this 
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area  (e.g.,  Graesser,  D’Mello,  &  Cade,  2010;  Hu,  Graesser,  and  Fowler,  in  press),  and  it 
remains  key  for  realizing  the  full  ADL  vision. 

G.  Conclusion 

In  short,  the  aforementioned  research  suggests  that  effective  use  of  ADL  technology 

•  Increases  instructional  effectiveness 

•  Reduces  time  needed  to  learn 

•  Ensures  that  all  students  learn 

•  Is  preferred  by  students 

•  Is  effective  and  efficient  for  distributing  instruction  anytime,  anywhere. 

Most  of  research  and  data  to  support  these  conclusions  have  been  available  for  some 
time.  The  usual  lag  between  research  findings  and  their  application  in  practice  is 
observable  here  as  elsewhere.  As  argued  first  by  Fletcher  (1992,  1997)  and  later  by 
Corbett  (2001),  ADL  technology  may  make  Scrivin’s  educational  imperative  and 
Bloom’s  tutorial  instruction  affordable. 

The  “Rule  of  Thirds”  that  emerges  from  empirical  evaluations  of  technology-based 
instruction  was  mentioned  earlier  in  this  report.  The  Rule  of  Thirds  is  strictly  a  statistical 
statement.  It  summarizes  a  large  body  of  empirical  findings,  but  it  does  not  directly 
address  cause  and  effect. 

An  often  quoted  statement  that  “The  best  current  evidence  is  that  media  are  mere 
vehicles  that  deliver  instruction  but  do  not  influence  student  achievement  any  more  than 
the  truck  that  delivers  our  groceries  causes  changes  in  our  nutrition”  (Clark,  1983,  p.  445) 
does  address  cause  and  effect.  This  point  of  view  seems  both  fair  and  unequivocal.  The 
presence  of  any  technology  is  no  guarantee  that  effective  instructional  content,  effective 
ways  to  present  it,  or  even  that  the  unique  strengths  of  the  technology  itself  will  be 
present  or  used.  On  the  other  hand,  the  absence  of  technology,  including  ADL  technol¬ 
ogy,  is  a  reasonable  guarantee  that  its  unique  functionalities  will  be  unavailable. 

Another  statement  to  consider  is  that  “If  you  don’t  have  a  gadget  called  a  ‘teaching 
machine,’  don’t  get  one.  Don’t  buy  one;  don’t  borrow  one;  don’t  steal  one.  If  you  have 
such  a  gadget,  get  rid  of  it.  Don’t  give  it  away,  for  someone  else  might  use  it.  .  .  .  If  you 
begin  with  a  device  of  any  kind,  you  will  try  to  develop  the  teaching  program  to  fit  that 
device”  (p.  478,  Gilbert,  1960). 

Gilbert  seems  both  right  and  wrong.  He  is  certainly  correct  in  suggesting  that 
instructional  designers  and  developers  who  adapt  a  “teaching  machine”  will  try  to  fit  the 
teaching  program  to  it.  However,  the  new  functionalities  such  a  device  makes  available 
motivate  its  adoption  and  instructional  adaptations  to  it. 
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It  is  less  certain  that  such  adaptations  are  to  be  avoided.  They  might  well  be  enthu¬ 
siastically  sought,  just  as  printed  textbooks  were  sought  long  ago.  If  properly  applied, 
technology  should  improve — if  not  revolutionize — the  effectiveness  and  efficiency  of 
teaching  programs.  It  is  up  to  researchers,  developers,  and  instructors — not  the  tech¬ 
nology  itself — to  see  that  it  does. 

Finally,  there  is  the  Columbus  effect.  In  keeping  with  technologies  that  made 
carriages  go  without  horses  and  telegraphs  transmit  without  wires,  the  Columbus  effect 
will  doubtless  apply  to  our  efforts  to  provide  tutorial  instruction  without  humans.  We 
envision  the  development  of  personal  learning  associates  and  are  building  toward  them. 
However,  just  as  Columbus  headed  for  the  East  Indies  and  ended  up  with  something 
entirely  unexpected,  we  may  end  up  with  something  as  unforeseen  and  different  from 
horseless  carriages  and  wireless  telegraph  as  automobiles  and  radio.  Nonetheless,  making 
education,  training,  and  problem  solving  aids  as  affordable  and  universally  accessible  as 
possible  seems  as  good  a  start  as  any. 


16 


Illustrations 


Figures 

Figure  1.  Individual  Tutoring  Compared  to  Classroom  Instruction . 4 

Figure  2.  Cost  Savings  ($2008)  in  Specialized  Skill  Training  Assuming  a  30% 

Reduction  in  Training  Time . 7 

Figure  3.  Some  Effect  Sizes  for  Studies  Comparing  Technology-Based  Instruction 

With  More  Conventional  Approaches . 8 

Tables 

Table  1.  Percent  Time  Savings  for  Technology-Based  Instruction . 6 

Table  2.  Maintenance  Performance  of  12  Air  Force  Avionics  Specialists  and 

12  APG  Technicians  Using  TOs  and  IMIS . 13 


A-l 


References 


Alpert,  D.  I.,  &  Bitzer,  D.  L.  (1970).  Advances  in  computer-based  education.  Science, 
167,  1582-1590. 

Angier,  B.,  Fletcher,  J.  D.,  &  Horowitz,  S.  (1991).  Interactive  Courseware  (ICW)  and  the 
cost  of  individual  training  (IDA  Paper  P-2567).  Alexandria,  VA:  Institute  for 
Defense  Analyses. 

Atkinson,  R.  C.  (1968).  Computerized  instruction  and  the  learning  process.  American 
Psychologist,  23,  225-239. 

Bloom,  B.  S.  (1984).  The  2  sigma  problem:  The  search  for  methods  of  group  instruction 
as  effective  as  one-to-one  tutoring.  Educational  Researcher,  13,  4-16. 

Camacho,  J.  (2009).  Joint  Knowledge  Online  Brief  to  the  Norwegian  Armed  Forces 
Norwegian  Defence  University  College  Education  Command.  ADL-conference 
2009.  Available: 

http://www.mil.no/multimedia/archive/00123/Joe_Camacho_plenary_123167a.pdf. 

Carbonell,  J.  R.  (December  1970).  AI  in  CAI:  An  artificial  intelligence  approach  to 

computer-assisted  instruction.  IEEE  Transactions  on  Man-Machine  Systems,  11  (4), 
190-202. 

Clark,  R.  E.  (1983).  Reconsidering  research  on  learning  from  media.  Review  of 
Educational  Research,  53,  445-459. 

Corbett,  A.  (2001).  Cognitive  computer  tutors:  Solving  the  two-sigma  problem.  In  M. 
Bauer,  P.  J.  Gmytrasiewicz,  &  J.  Vassileva  (Eds.),  User  modeling  (pp.  137-147). 
Berlin:  Springer- Verlag. 

Corno,  L.,  &  Snow,  R.  E.  (1986).  Adapting  teaching  to  individual  differences  among 
learners.  In  M.  C.  Wittrock  (Ed.),  Handbook  of  research  on  teaching,  (3rd  ed., 
pp.  605-629).  New  York:  Macmillan. 

Coulson,  J.  E.  (Ed.).  (1962).  Programmed  learning  and  computer-based  instruction.  New 
York,  NY:  John  Wiley. 

Dochy,  F.,  Segers,  M.,  &  Buehl,  M.  (1999).  The  relation  between  assessment  practices 
and  outcomes  of  studies:  The  case  of  research  on  prior  knowledge.  Review  of 
Educational  Research,  69(2),  145-186. 

Dodds,  P.  V.  W.,  &  Fletcher,  J.  D.  (2004).  Opportunities  for  new  “smart”  learning 
environments  enabled  by  next  generation  web  capabilities.  Journal  of  Education 
Multimedia  and  Hypermedia,  13  (4),  391-404. 

Fletcher,  J.  D.  (1991).  Effectiveness  and  cost  of  interactive  videodisc  instruction. 
Machine  Mediated  Learning,  3,  361-385. 


B-l 


Fletcher,  J.  D.  (1992).  Individualized  systems  of  instruction.  In  M.  C.  Alkin  (Ed.), 
Encyclopedia  of  educational  research  (6th  ed.,  pp.  613-620).  New  York,  NY: 
Macmillan. 

Fletcher,  J.  D.  (1997).  What  have  we  learned  about  computer  based  instruction  in  military 
training?  In  R.  J.  Seidel  &  P.  R.  Chatelier  (Eds.),  Virtual  reality,  training ’s  future ? 
(pp.  169-177).  New  York,  NY:  Plenum  Publishing. 

Fletcher,  J.D.  (2004).  Technology,  the  Columbus  effect,  and  the  third  revolution  in 
learning.  In  M.  Rabinowitz,  F.  C.  Blumberg,  &  H.  Everson  (Eds.),  The  design  of 
instruction  and  evaluation:  Affordances  of  using  media  and  technology  (pp.  139— 
157).  Mahwah,  NJ:  Lawrence  Erlbaum  Associates. 

Fletcher,  J.  D.,  &  Atkinson,  R.  C.  (1972).  An  evaluation  of  the  Stanford  CAI  program  in 
initial  reading  (grades  K  through  3).  Journal  of  Educational  Psychology,  63,  597- 
602. 

Fletcher,  J.  D.,  &  Johnston,  R.  (2002).  Effectiveness  and  cost  benefits  of  computer-based 
aids  for  maintenance  operations.  Computers  in  Human  Behavior,  18,  717-728. 

Fletcher,  J.  D.,  &  Suppes,  P.  (1972).  Computer-assisted  instruction  in  reading: 

Grades  4-6.  Educational  Technology,  12,  45-49. 

Fletcher,  J.  D.,  &  Tobias,  S.  (2005).  The  multimedia  principle.  In  R.  E.  Mayer  (Ed.),  The 
Cambridge  handbook  of  multimedia  learning  (pp.  117-133).  New  York,  NY : 
Cambridge  University  Press. 

Fowler,  B.  T.  (1980).  The  effectiveness  of  computer-controlled  videodisc-based  training. 
(Doctoral  dissertation,  University  of  Iowa,  1980).  University  Microfilms 
No.  8114254 

Gafford,  W.,  &  Heller,  M.  (in  press).  ADL  and  enterprise  content  management.  In  R.  A. 
Wisher  &  B.  H.  Khan  (Eds.),  Learning  on  demand:  ADL  and  the  future  of  e- 
learning.  Washington  DC:  Department  of  Defense. 

Galanter,  E.  (Ed.).  (1959).  Automatic  teaching:  The  state  of  the  art.  New  York:  John 
Wiley  and  Sons. 

Gallager,  P.  S.  (in  press).  The  development  of  SCORM.  In  R.  A.  Wisher  &  B.  H.  Khan 
(Eds.),  Learning  on  demand:  ADL  and  the  future  of  e-learning.  Washington  DC: 
Department  of  Defense. 

Gettinger,  M.  (1984).  Individual  differences  in  time  needed  for  learning:  A  review  of  the 
literature.  Educational  Psychologist,  19,  15-29. 

Gilbert,  T.  F.  (1960).  On  the  relevance  of  laboratory  investigation  of  learning  to  self- 
instructional  programming.  In  A.  A.  Lumsdaine  &  R.  Glaser  (Eds.),  Teaching 
machines  and  programmed  learning:  A  source  book  (pp.  475-485).  Washington, 
DC:  National  Education  Association  of  the  United  States. 

Graesser,  A.  C.,  &  Person,  N.  K.  (1994).  Question  asking  during  tutoring.  American 
Educational  Research  Journal,  31, 104-137. 


B-2 


Graesser,  A.  C.,  D’Mello,  S.,  &  Cade,  W.  (2010).  Instruction  based  on  tutoring.  In  R.  E. 
Mayer  &  P.  A.  Alexander,  (Eds.),  Handbook  of  research  and  learning  on 
instruction  (Chapter  20).  New  York:  Routledge  Press. 

Greiner,  J.  M.  (1991).  Interactive  multimedia  instruction:  What  do  the  numbers  show?  In 
Proceedings  of  the  Ninth  Annual  Conference  on  Interactive  Instruction  Delivery 
(pp.  100-104).  Warrenton,  VA:  Society  for  Applied  Learning  Technology. 

Hu,  X.,  Graesser,  A.,  &  Fowler,  D.  (in  press).  ADL  and  intelligent  tutoring  systems.  In 
R.  A.  Wisher  &  B.  H.  Khan  (Eds.),  Learning  on  demand:  ADL  and  the  future  of  e- 
learning.  Washington  DC:  Department  of  Defense. 

Kilgour,  F.  G.  (1998).  The  evolution  of  the  book.  New  York,  NY:  Oxford  University 
Press. 

Kulik,  J.  A.  (1994).  Meta-analytic  studies  of  findings  on  computer-based  instruction.  In 
E.  L.  Baker  &  H.F.  O’Neil,  Jr.  (Eds.),  Technology  Assessment  in  Education  and 
Training  (pp.  9-33).  Hillsdale,  NJ:  Lawrence  Erlbaum  Associates. 

Lammon,  L.  (in  press).  The  ADL  registry:  A  technology  for  finding  and  reusing  e- 

leaming  content.  In  R.  A.  Wisher  &  B.  H.  Khan  (Eds.),  Learning  on  demand:  ADL 
and  the  future  of  e-learning.  Washington  DC:  Department  of  Defense. 

Mayer,  R.  E.  (Ed.).  (2005).  The  Cambridge  handbook  of  multimedia  learning.  New  York, 
NY :  Cambridge  University  Press. 

McKinnon,  D.  H.,  Nolan,  C.  J.  P.,  &  Sinclair,  K.  E.  (2000).  A  longitudinal  study  of 

student  attitudes  toward  computers:  Resolving  an  attitude  decay  paradox.  Journal  of 
Research  on  Computing  in  Education,  32,  325-335. 

Military  Manpower  Training  Report.  (2002).  Washington,  DC:  Defense  Manpower  Data 
Center,  U.S.  Department  of  Defense. 

Military  Personnel  Composite  Standard  Pay  and  Reimbursement  Rates.  (2009). 
Washington,  DC:  Office  of  the  Under  Secretary  of  Defense  (Comptroller). 

Available:  http://comptroller.defense.gov/rates/. 

Noja,  G.  P.  (1987).  New  frontiers  for  computer-aided  training.  In  R.  J.  Seidel  &  P.  D. 
Weddle  (Eds.),  Computer-based  instruction  in  military  environments  (pp.  215-229). 
New  York:  Plenum  Press. 

Orlansky,  J.,  &String,  J.  (1977).  Cost  effectiveness  of  computer-based  instruction  in 
military  training  (IDA  Paper  P-1375).  Arlington,  VA:  Institute  for  Defense 
Analyses. 

Scriven,  M.  (1975).  Problems  and  prospects  for  individualization.  In  H.  Talmage  (Ed.), 
Systems  of  individualized  education  (pp.  199-210).  Berkeley,  CA:  McCutchan. 

Sleeman,  D.,  &  Brown,  J.  S.  (Eds.).  (1982).  Intelligent  tutoring  systems.  New  York,  NY: 
Academic  Press. 

Suppes,  P.  (1966).  The  uses  of  computers  in  education.  Scientific  American,  215,  206- 
220. 


B-3 


Suppes,  P.,  Fletcher,  J.  D.,  &  Zanotti,  M.  (1975).  Performance  models  of  American 
Indian  students  on  computer-assisted  instruction  in  elementary  mathematics. 
Instructional  Science,  4,  303-313. 

Suppes,  P.,  Fletcher,  J.  D.,  &  Zanotti,  M.  (1976).  Models  of  individual  trajectories  in 
computer-assisted  instruction  for  deaf  students.  Journal  of  Educational  Psychology, 
68,  117-127. 

Thomas,  D.  L.  (1995).  Integrated  Maintenance  Information  System:  User  Field 

Demonstration  and  Test  Executive  Summary  (Technical  Report  AL/HR-TR-1995- 
0033).  Wright-Patterson  Air  Force  Base,  OH:  Human  Resources  Directorate, 
Logistics  Research  Division.  DTIC/NTIS  ADA  293  250 

Teitelbaum,  D.,  &  Orlansky,  J.  (1996).  Costs  and  benefits  of  the  Integrated  Maintenance 
Information  System  (IMIS)  (IDA  Paper  P-3173).  Alexandria,  VA:  Institute  for 
Defense  Analyses.  DTIC/NTIS  ADA  312  884 

Tobias,  S.  (1989).  Another  look  at  research  on  the  adaptation  of  instruction  to  student 
characteristics .  Educational  Psychologist,  24,  213-227. 

Verano,  M.  (1987).  Achievement  and  retention  of  Spanish  presented  via  videodisc  in 
linear,  segmented,  and  interactive  modes.  (Doctoral  dissertation,  University  of 
Texas  at  Austin,  1987).  DTIC  ADA  1 85  893 

Wetzel-Smith,  S.  K.,  &.  Wulfeck,  W.  H.  (2010).  Training  incredibly  complex  tasks.  In 
J.  V.  Cohn  &  P.  E.  O’Connor  (Eds.),  Performance  enhancement  in  high-risk 
environments  (pp.  74-89).  Westport,  CT:  Praeger. 

Wisher,  R.  A.  (in  press).  Introduction  to  ADL.  In  R.  A.  Wisher  &  B.  H.  Khan  (Eds.), 
Learning  on  demand:  ADL  and  the  future  of  e-learning.  Washington  DC: 
Department  of  Defense. 

Wulfeck,  W.  H.,  &  Wetzel-Smith,  S.  K.  (2008).  Use  of  visualization  techniques  to 
improve  high-stakes  problem  solving.  In  E.  Baker,  J.  Dickieson,  W.  Wulfeck,  & 

H.  F.  O’Neil  (Eds.),  Assessment  of  problem  solving  using  simulations  (pp.  223- 
238).  Florence,  KY:  Taylor  and  Francis-Lawrence  Erlbaum  Associates. 


B-4 


Abbreviations 


ADL 

Advanced  Distributed  Learning 

APG 

Airplane  General 

CBI 

computer-based  instruction 

COR DR A 

Content  Object  Repository  Registration/Resolution 
Architecture 

DTIC 

Defense  technical  Information  Center 

ICW 

Interactive  Courseware 

IETM 

Interactive  Electronic  Technical  Manual 

IMI 

interactive  multimedia  instruction 

IMIS 
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